Motion capture system

ABSTRACT

Systems and methods for motion capture are disclosed. In one arrangement a plurality of cameras are provided and a processing unit is configured to perform processing using image data from each of the plurality of cameras. Each of the cameras comprises an accelerometer and the processing unit is configured to perform processing that is dependent on an output from the accelerometers. In another arrangement each of the cameras comprises a temperature sensor and the processing unit is configured to perform processing that is dependent on an output from the temperature sensors.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a U.S. national stage application ofInternational Patent Application No. PCT/GB2016/051130, filed Apr. 22,2016, which claims the benefit of priority to GB Patent Application No.1509993.0, filed Jun. 9, 2015, the entireties of which are incorporatedherein by reference.

The present invention relates to a motion capture system and a method ofperforming motion capture, in which a plurality of cameras are eachprovided with an accelerometer and/or a temperature sensor.

A motion capture system typically comprises a plurality of camerasconfigured to monitor a target region in which the movements of atarget, for example an actor, are to be captured. The captured movementsmay be used to animate a digital character in a film or computer gamebased on the actor's movements, for example. Typically, in such systemseach of the cameras will need to be mounted at separate locations anddirected towards the target region. The cameras will also need to becalibrated and connected to a processing unit such as a desktop orlaptop computer, or a tablet. Setting up such motion capture systems canbe time consuming, requiring for example multiple return trips for anoperator between the processing unit and each of the cameras and/or theinvolvement of multiple people. For example, one person might interactdirectly with the processing unit while one or more other people maymove around the room individually adjusting and checking each of thecameras while communicating with the person at the processing unit. Theperson at the processing unit may provide feedback to the movingoperator about the image being captured by each of the cameras, as themoving operator moves from one camera to the next.

Motion capture systems may be sensitive to environmental conditions andto disturbances. For example, deviations in the quality of a cameramounting (e.g. due to knocks, thermal expansion/contraction, bending,etc.) may cause the position or orientation of the camera to change overtime. This can have a negative impact on the motion capture processand/or require frequent and time consuming recalibrations.

A further challenge with existing systems is that temperature variationswithin the cameras themselves may alter the performance of the cameras,for example by causing differences in the optical path from a lensthrough to an imager of the camera. These effects can be mitigated insome circumstances by calibrating the cameras after the temperature hasreached a steady state. However, judging when to perform suchcalibration is difficult. Users may be advised to wait a predeterminedperiod after start up of the system before performing camera calibrationand beginning use of the system. However, this can cause inconvenientdelay. Furthermore, such an approach is not effective where a steadystate temperature is not reached reproducibly or where the temperaturevaries during use of the system, for example due to variations in theactivity of individual cameras or in the temperature of the surroundingenvironment.

It is an object of the invention to provide a motion capture system andmethod that at least partially address one or more of the challengesdiscussed above.

According to an aspect of the invention, there is provided a motioncapture system comprising: a plurality of cameras; and a processing unitconfigured to perform processing using image data from each of theplurality of cameras, wherein each of the cameras comprises anaccelerometer and the processing unit is configured to performprocessing that is dependent on an output from the accelerometers.

Thus, a system is provided in which individual cameras can respond tooutput from an accelerometer provided in each camera. This capabilityenhances flexibility in how the motion capture system can be set up andmaintained, facilitating in particular procedures which involve closeproximity of a user with individual cameras. For example, the system canbe configured to use the accelerometer outputs to allow a user to selectindividual cameras by causing perturbations in the accelerometeroutputs, e.g. by tapping or double tapping a camera. The accelerometersmay alternatively or additionally be used to detect unwanted deviationsin the orientation of cameras, thereby to prompt investigatory actionand/or recalibration procedures.

According to an aspect of the invention, there is provided a motioncapture system comprising: a plurality of cameras; and a processing unitconfigured to perform processing using image data from each of theplurality of cameras, wherein each of the cameras comprises atemperature sensor and the processing unit is configured to performprocessing that is dependent on an output from the temperature sensors.

Thus, a system is provided in which individual cameras can respond tovariations in a temperature of each camera. This capability facilitatesdetection of when a steady state temperature has been reached after apowering on of the camera, making it possible for calibration proceduresto be started more reliably in the steady state regime, and potentiallyearlier than would be possible using other approaches. The capabilityalso makes it possible to respond to variations in temperature that mayoccur at times other than shortly after powering on, for example due tochanges in the environmental temperature.

According to an alternative aspect of the invention, there is provided amethod of performing motion capture using a plurality of cameras,wherein each of the cameras comprises an accelerometer, the methodcomprising: performing processing at a processing unit that is dependenton an output from the accelerometers.

According to an alternative aspect of the invention, there is provided amethod of performing motion capture using a plurality of cameras,wherein each of the cameras comprises a temperature sensor, the methodcomprising: performing processing at a processing unit that is dependenton an output from the temperature sensor.

Embodiments of the invention will now be described, by way of exampleonly, with reference to the accompanying drawings in which correspondingreference symbols indicate corresponding parts, and in which:

FIG. 1 depicts a motion capture system;

FIG. 2 depicts a top view of a plurality of cameras of a motion capturesystem directed towards a target region; and

FIG. 3 is a schematic side view of a single one of the plurality ofcameras of FIG. 2.

In an embodiment, an example of which is shown in FIGS. 1-3, there isprovided a motion capture system 2 comprising a plurality of cameras 4.A processing unit 6 is configured to perform processing using image datareceived from each of the plurality of cameras 4. The processing maycomprise processing associated with performing motion capture forexample. In this context, motion capture is understood to encompass atleast any process or technique of recording patterns of movementdigitally, for example for recording of an actor's movements for thepurpose of animating a digital character in a film or computer game.Optionally, one or more of the plurality of cameras 4 may be configuredto perform processing that contributes to the motion capture procedure.For example a camera may pre-process image data captured by the camerabefore sending data to the processing unit. The processing unit 6 maytake various forms. For example, the processing unit 6 may comprise acomputer (e.g. a desktop or laptop PC), a tablet computer, a smarttelephone, or any combination of these in communication with each other.The processing unit 6 may comprise a display 8. The processing unit 6may further comprise processing hardware 10 (e.g. CPU, memory, etc.),and an interface 12 allowing a user to interact with the processing unit6. The interface may comprise a touch screen, keyboard, mouse, or otherinput device.

In an embodiment each of the cameras 4 comprises an accelerometer 22. Anexample of such a camera is shown in FIG. 3. In embodiments of this typethe processing unit 6 is configured to perform processing that isdependent on an output from the accelerometers 22.

In an embodiment the processing unit 6 is configured to select one ofthe cameras 4 based on an output from the accelerometer 22 of thatcamera. The processing unit 6 is further configured to perform aselected-camera processing which depends on which of the cameras 4 hasbeen selected. The selected-camera processing may comprise variousdifferent functionalities associated with the selected camera. In anembodiment, the selected-camera processing comprises displaying data,for example on the display 8, from the selected camera 4. Thisfunctionality therefore makes it possible for a user to view dataassociated with a particular camera merely by causing an output from theaccelerometer to vary in a particular way (e.g. by tapping or otherwisemanipulating the camera 4).

In an embodiment, the selected-camera processing comprises displayingdata from a plurality of the cameras and providing visual informationindicating which of the displayed data corresponds to the selectedcamera. For example, the processing unit 6 may be configured to displayimage data captured by more than one of the cameras 4 in a correspondingplurality of different windows in the display 8. The selected camera maybe indicated by highlighting the window corresponding to the selectedcamera, or by bringing the window to the front relative to otherwindows, for example.

Alternatively or additionally, the selected-camera processing maycomprise displaying data from the selected camera and not displayingdata from any other camera. In this way, manipulation of the camera tocause a characteristic change in the output from the accelerometer 22 inthat camera can be used to cause the display 8 to switch betweendisplaying data from different cameras.

The above-described functionality may be particularly convenient in thecase where the processing unit 6 comprises a portable unit, such as alaptop or tablet. In this case, a user can carry the portable unit fromone camera 4 to another camera 4 in the system and conveniently selectto view data associated with a camera that he is currently next tomerely by causing the output from the accelerometer to change (e.g. bytapping the camera 4). This functionality avoids the user having tomanually interact with the portable unit itself in order to select acamera. This approach may save considerable time and effort,particularly in the case where many different cameras 4 are provided.

In an embodiment, the displayed data of the selected-camera processingcomprises image data captured by the selected camera 4. Alternatively oradditionally, the displayed data comprises configuration data of theselected camera. The configuration data may comprise any operatingparameters or other parameters of the camera 4. In an embodiment, theselected-camera processing comprises providing a selected camerainterface, for example as part of the display 8, for example a touchscreen active part of the display 12, which allows a user to control theselected camera 4. Thus, by tapping or otherwise manipulating the camera4 to provide a change in the output from the accelerometer 22, the useris able conveniently to bring up an interface which allows him tocontrol the selected camera 4 via the processing unit 6 (e.g. laptop ortablet).

The selected-camera interface may allow control of a plurality ofcameras 4 and provide a visual indication indicating which of thecameras 4 is the selected camera 4. This approach allows the user toview configuration data or other relevant data from other cameras whilecontrolling the selected camera 4. This may be useful where the datafrom other cameras is relevant for comparison or reference purposes.

In an embodiment, the selected-camera interface is configured to allowcontrol of the selected camera 4 and no other camera 4. Various otherfunctionalities associated with the selected camera 4 may be provided bythe selected-camera interface.

In an embodiment, the processing unit 6 is configured to select one ofthe cameras 4 in response to a perturbation in the output of theaccelerometer 22 (e.g. an amplitude of the perturbation, proportional toa magnitude of an acceleration, or a duration of the perturbation) fromthat camera 4 that is larger than a predetermined threshold. Theperturbation may be caused for example by a user of the system tappingthe camera 4 lightly.

Alternatively or additionally, the processing unit 6 may be configuredto select one of the cameras 4 when an output from the accelerometer 22from that camera 4 matches a predetermined signature output. Thepredetermined signature output may correspond to an output consistentwith a user tapping the camera with a finger, or any other manipulationof the camera by a user which does not disturb the mounting of thecamera 4. Typically, therefore, the processing unit 6 is configured todetect relatively light interaction between the user and the camera 4,such as tapping (single or multiple, such as double tapping), or othersimilar, light interactions. The predetermined signature output may bedetermined by recording the response of the camera accelerometer 22 toone or more types of envisaged interaction. For example, the response toa variety of single finger taps could be recorded and an averageresponse could be used as a predetermined signature output. When a usertaps the camera 4 in use, the response of the accelerometer 22 can becompared to the recorded average response and if the two responses aresufficiently similar it can be deduced that there is a match between theoutput from the accelerometer and the predetermined signature output.Various aspects of the accelerometer output could be used for thecomparison, including for example the duration of the perturbation inthe output, the amplitude of the perturbation of the output, or thevariation with time of the amplitude of the perturbation. Using thevariation with time of the amplitude of the perturbation may beparticularly appropriate for example where the interaction by the usertakes a more complex form, for example a double tap. Here, it may beappropriate for example to detect a pair of local maxima in theperturbation of the accelerator output 22 and/or the time separation ofthe maxima in order to determine whether the user interaction matchesthe predetermined signature output for a double tap.

In an embodiment, the processing unit 6 is configured to responddifferently according to the direction of a perturbation in the outputfrom the accelerometer 22. For example, the processing unit 6 may beconfigured to perform a first direction-selected processing when thecamera 4 is tapped on one side (and thus in a first direction, forexample in an inward direction substantially perpendicular to the firstside) and a second direction-selected processing, different from thefirst direction-selected processing, when the camera 4 is tapped on asecond side (and thus in a second direction, different from the firstdirection, for example in an inward direction substantiallyperpendicular to the second side). For example, the firstdirection-selected processing may comprise displaying image datacaptured by the camera and the second direction-selected processing maycomprise displaying configuration data of the camera.

FIG. 2 depicts how four cameras 4 of a motion capture system 2 may beconfigured so as to be directed towards a target region 16. Theprovision of four cameras 4 is exemplary. Fewer than four or more thanfour cameras 4 may be provided according to requirements. Typically,each of the cameras 4 will be a physically separate unit. Each camera 4is typically mountable independently of the other cameras 4. In theexample shown, the cameras 2 are mounted to separate mounting supports14.

The cameras 4 may be configured to interact with the processing unit 6in various ways. Wires may be provided between each of the cameras 4 andthe processing unit 6, for example. Alternatively or additionally,wireless data transfer may be possible between the cameras 4 and theprocessing unit 6.

In the above-described embodiments, accelerometers 22 in the cameras 4are used to provide a convenient way for a user to select a camera thathe is in close proximity with and/or to select functionality associatedwith that camera. However, the accelerometers 22 may be used for otherpurposes, in addition to the functionalities described above, or as analternative to the functionalities described above.

For example, in an embodiment the processing unit 6 is configured todetect a change in an orientation of one or more of the cameras 4 basedon an output from the accelerometers 22. This may be achieved bydetecting a change in the direction of gravity relative to the camera 4.This is illustrated schematically in the side view of FIG. 3. Arrow 26indicates the direction of gravity relative to the accelerometer 22.Gravity acts as an acceleration and is therefore detected by theaccelerometer 22. The accelerometer 22 is able to detect the directionof gravity 26 relative to a reference axis 28 of the camera 4. In theexample shown in FIG. 3, this is used to allow the accelerometer 22 todetermine an angle 24 of inclination of the camera 4 relative to thevertical. Typically, when the motion capture system 2 is set up, one ormore of the cameras 4 will be mounted fixedly and then calibrated beforethe motion capture system is used to capture motion in the target region16. Any deviation in the position or orientation of a camera 4 after thecalibration process has finished will lead to a reduction in the qualityof the image capture process. Such deviations may occur for example dueto knocks to the camera 4 or mounting 14, variations in temperature andassociated thermal expansion/contraction in the camera 4 or mounting 14,or yielding or bending of the mounting 14, etc. The inventors haverecognised that a wide range of such deviations will involve a change inorientation of the camera 4 relative to the direction of gravity andthat this can be detected by an accelerometer 22. Thus, theaccelerometer 22 provides a convenient and accurate way for theprocessing unit 6 to detect when there has been a deviations in thecamera mounting. Even changes which occur slowly over time, such as whena mounting is gradually bending or changing due to long time-scaletemperature variations, etc., the accelerometer 22 can still detectchanges in the orientation by comparing an absolute measurement of theorientation at a camera 4 with an absolute recording of the orientationof the same camera 4 at a previous time. For example, when the system isleft overnight, a comparison can be made between the outputs from theaccelerometers 22 in the morning compared with the outputs recorded fromthe previous day. Any significant variations (e.g. variations that arelarger than a predetermined threshold) will indicate that a camera hasbeen disturbed significantly and that checking of the mounting and/orrecalibration is therefore necessary.

In the above-described embodiments, each of the cameras 4 is providedwith an accelerometer 22. In these and other embodiments each of thecameras 4 may alternatively or additionally comprise one or more othersensors which provide useful information about the state of the camera.

In an embodiment, each of the cameras 4 comprises a temperature sensor18. The temperature sensor 18 may be provided instead of or in additionto the accelerometer 22. The processing unit 6 is configured in suchembodiments to perform processing that is dependent on an output fromthe temperature sensor 18. In an embodiment the processing unit 6 isconfigured to provide a visual indication of the temperature of one ormore of the cameras 4. The visual indication may be provided on thedisplay 12 of the processing unit 6 for example. Alternatively oradditionally, a visual indication of the temperature may be provided atthe individual cameras 4. In an embodiment, the temperature sensor 18 ofeach camera 4 is mounted on computer processing hardware 20, such as aPCB, within the camera 4. The temperature sensor 18 in such anembodiment is therefore configured to measure the temperature of thecomputer processing hardware 20. Typically, when a camera 4 is poweredon power is supplied to the computer processing hardware 20 within thecamera 4, which causes the hardware 20 to heat up. The heating can leadto changes in operating characteristics of the computer processinghardware 20 and/or in characteristics of other elements of the camera.For example, heat can be conducted from the computer processing hardware20 to optical or mechanical elements of the camera, for example betweena lens of the camera 4 and an image capture unit configured to transformlight received at pixels thereof into an electrical signal representingthe image captured. By measuring the temperature of the computerprocessing hardware 20 it is possible to detect for example when thetemperature of the hardware 20 reaches a steady state. At this point,further thermal-driven changes in the properties of the camera 4 arelikely to be very small. A user can therefore safely commencecalibration procedures at this point in the knowledge that subsequenttemperature variations are unlikely to cause significant deviation fromthe calibrated state. Relative to alternative approaches in which a usersimply waits for a predetermined type which is chosen so as to be safelylonger than any temperature settling period, the approach of the presentembodiment allows a user to start the calibration procedure earlierand/or at a point which is more reliably in the steady state temperatureregime.

The invention claimed is:
 1. A motion capture system comprising: aplurality of cameras; and a processing unit configured to performprocessing using image data from each of the plurality of cameras,wherein each of the cameras comprises an accelerometer and theprocessing unit is configured to perform processing that is dependent onan output from the accelerometers; wherein the processing unit isconfigured to perform a first direction-selected processing associatedwith a camera when a perturbation in the acceleration at the camera in afirst direction is detected and to perform a second direction-selectedprocessing associated with the camera when a perturbation in theacceleration at the camera in a second direction is detected, the firstand second directions being different from each other, and the first andsecond direction-selected processings being different from each other.2. The system of claim 1, wherein the processing unit is configured to:select one of the cameras based on an output from the accelerometer; andperform a selected-camera processing which depends on which of thecameras has been selected.
 3. The system of claim 2, wherein theselected-camera processing comprises displaying data from the selectedcamera.
 4. The system of claim 3, wherein the selected-camera processingcomprises displaying data from a plurality of the cameras and providingvisual information indicating which of the displayed data corresponds tothe selected camera.
 5. The system of claim 3, wherein theselected-camera processing comprises displaying data from the selectedcamera and not displaying data from any other camera.
 6. The system ofclaim 2, wherein the displayed data comprises image data captured by theselected camera.
 7. The system of claim 2, wherein the displayed datacomprises configuration data of the selected camera.
 8. The system ofclaim 2, wherein the selected-camera processing comprises providing aselected-camera interface to a user that allows a user to control theselected camera.
 9. The system of claim 8, wherein the selected-camerainterface allows control of a plurality of the cameras and provides avisual indication indicating which of the cameras is the selectedcamera.
 10. The system of claim 8, wherein the selected-camera interfaceallows control of the selected camera and no other camera.
 11. Thesystem of claim 2, wherein the processing unit is configured to selectone of the cameras in response to a perturbation in the output of theaccelerometer from that camera that is larger than a predeterminedthreshold.
 12. The system of claim 2, wherein the processing unit isconfigured to select one of the cameras when an output of theaccelerometer from that camera matches a predetermined signature output.13. The system of claim 12, where the predetermined signature outputcorresponds to an output consistent with a user tapping the camera witha finger.
 14. The system of claim 12, wherein the predeterminedsignature output corresponds to an output consistent with a user tappingthe camera a predetermined multiplicity of times.
 15. The system ofclaim 1, wherein each of the cameras is a physically separate unit andis mountable independently of the other cameras.
 16. The system of claim1, wherein the processing unit is provided in a portable computingdevice configured to communicate wirelessly with each of the cameras.17. The system of claim 1, wherein the processing unit is configured todetect a change in an orientation of any of the cameras based on anoutput from the accelerometers.
 18. The system of claim 17, wherein thedetection of a change in an orientation of a camera is performed bydetecting a change in the direction of gravity relative to that camera.19. A method of performing motion capture using a plurality of cameras,wherein each of the cameras comprises an accelerometer, the methodcomprising: performing processing at a processing unit that is dependenton an output from the accelerometers, wherein the processing unit isconfigured to perform a first direction-selected processing associatedwith a camera when a perturbation in the acceleration at the camera in afirst direction is detected and to perform a second direction-selectedprocessing associated with the camera when a perturbation in theacceleration at the camera in a second direction is detected, the firstand second directions being different from each other, and the first andsecond direction-selected processings being different from each other.20. The method of claim 19, wherein the processing comprises detecting achange in an orientation of any of the cameras based on an output fromthe accelerometers.
 21. The method of claim 20, further comprisingfixedly mounting each of the cameras so as to be directed towards atarget scene and using the detected change in an orientation of a camerato detect a deviation in the mounting of the camera.